Add ComputeDomain for running multi-node workloads #225
Conversation
klueska
force-pushed
the
add-multi-node-crd
branch
10 times, most recently
from
38065b4 to
3e51cd8
Compare
klueska
force-pushed
the
add-multi-node-crd
branch
20 times, most recently
from
4ce9bdb to
0d435d8
Compare
Signed-off-by: Kevin Klues <[email protected]>
Signed-off-by: Kevin Klues <[email protected]>
Signed-off-by: Kevin Klues <[email protected]>
Signed-off-by: Kevin Klues <[email protected]>
Signed-off-by: Kevin Klues <[email protected]>
Signed-off-by: Kevin Klues <[email protected]>
Signed-off-by: Kevin Klues <[email protected]>
Signed-off-by: Kevin Klues <[email protected]>
Signed-off-by: Kevin Klues <[email protected]>
For now just mark one well-known erro as permanent. Future commits will abstract this better and mark more errors as permanaent. Signed-off-by: Kevin Klues <[email protected]>
Signed-off-by: Kevin Klues <[email protected]>
Signed-off-by: Kevin Klues <[email protected]>
Signed-off-by: Kevin Klues <[email protected]>
Signed-off-by: Kevin Klues <[email protected]>
Signed-off-by: Kevin Klues <[email protected]>
Signed-off-by: Kevin Klues <[email protected]>
Signed-off-by: Kevin Klues <[email protected]>
klueska
force-pushed
the
add-multi-node-crd
branch
from
1e6a587 to
a45c238
Compare
Signed-off-by: Kevin Klues <[email protected]>
Signed-off-by: Kevin Klues <[email protected]>
Signed-off-by: Kevin Klues <[email protected]>
Signed-off-by: Kevin Klues <[email protected]>
klueska
force-pushed
the
add-multi-node-crd
branch
2 times, most recently
from
df90001 to
764c3c7
Compare
| // ComputeDomainSpec provides the spec for a ComputeDomain. | ||
| type ComputeDomainSpec struct { | ||
| NumNodes int `json:"numNodes"` | ||
| Channel *ComputeDomainChannelSpec `json:"channel"` |
There was a problem hiding this comment.
Should channel be optional thinking on non imex use cases in the future, I know currently we are solely focused on imex support, but if we want to carry on the concept of computeDomain, we might face clusters without imex (channels)
| @@ -0,0 +1,89 @@ | |||
| /* | |||
| * Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved. | |||
There was a problem hiding this comment.
maybe like some clothing brands do Since 1987* but I am not lawyer, maybe the number on the license header has a deeper legal meaning
| @@ -0,0 +1,49 @@ | |||
| /* | |||
| * Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved. | |||
Signed-off-by: Kevin Klues <[email protected]>
klueska
force-pushed
the
add-multi-node-crd
branch
from
764c3c7 to
474f968
Compare
| tail -f /dev/null & wait | ||
| fi | ||
| /usr/bin/nvidia-imex -c /etc/nvidia-imex/config.cfg | ||
| tail -n +1 -f /var/log/nvidia-imex.log & wait |
There was a problem hiding this comment.
Discussed in a sync meeting a while ago: here we give up control of the IMEX daemon process (do we? how does errexit behave when a daemonized process exits non-zero?). In any case, for robustness and debuggability it will be good to actively monitor the health of the IMEX daemon process (polling the process, or better: getting a health signal actively and straight from the process). I'd like to look into that at some point, after merge.
There was a problem hiding this comment.
This is still a problem. If the daemon crashes we will not exit (but the liveness probe will eventually fail and the pod will be restarted). We should make it more robust as a followup (probably by not doing everything in bash but instead writing a small go utility).
No description provided.